Skip to content

feat(video): replay recorded tracking data via VideoSourceInput.trackingDataUrl#10

Merged
msilivonik-sc merged 3 commits into
Snapchat:mainfrom
gsasson-sc:gsasson/video-tracking-data-replay
Jun 29, 2026
Merged

feat(video): replay recorded tracking data via VideoSourceInput.trackingDataUrl#10
msilivonik-sc merged 3 commits into
Snapchat:mainfrom
gsasson-sc:gsasson/video-tracking-data-replay

Conversation

@gsasson-sc

Copy link
Copy Markdown
Contributor

What

Adds an optional trackingDataUrl to VideoSourceInput. When set, the URL is fetched and passed to @snap/camera-kit's createVideoSource as trackingData, so the lens is driven by recorded tracking (camera pose, etc.) instead of live tracking.

Why

Enables previewing world-facing lenses from a recorded environment (e.g. a Lens Studio .td clip) — the core SDK already supports createVideoSource(video, { trackingData }), but the React wrapper had no way to pass it.

Changes

  • types.ts — add VideoSourceInput.trackingDataUrl?: string (documented).
  • internal/sourceUtils.ts — fetch the buffer in createCameraKitVideoSource and forward it to createVideoSource; reject on fetch failure.
  • internal/sourceUtils.test.ts — tests for pass-through, omitted, and fetch-failure.

trackingDataUrl (URL, not a raw buffer) keeps VideoSourceInput a serializable/hashable descriptor, consistent with how url is already handled and with useApplySource's source memoization.

Test

npm test (292 passing, incl. 3 new), npm run build, npm run typecheck, Prettier — all clean.

🤖 Generated with Claude Code

…ingDataUrl

Add optional `trackingDataUrl` to VideoSourceInput. When set, the URL is
fetched and passed to `createVideoSource` as `trackingData`, so the lens is
driven by recorded tracking (camera pose, etc.) instead of live tracking —
enabling recorded-environment previews of world-facing lenses.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@msilivonik-sc

msilivonik-sc commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Clarifying the concern here: if trackingDataUrl is provided, I think the tracking data should remain required and createCameraKitVideoSource should reject if it cannot be fetched. The issue is the current ordering starts playback before the sidecar is loaded. With a slow .td fetch, the hidden looping video can advance before Camera Kit receives the source, which can offset replay tracking from the video timeline. If the fetch fails, the promise rejects after playback has already started and the provider has no created source to clean up, so the video can keep playing in the background.

I would fetch the tracking data before videoInput.play() (or start the fetch earlier in parallel with video loading, but still avoid playback until the tracking data is ready). Only if the desired API contract is “best effort, fall back to live tracking” would I make the tracking data non-essential, but that seems different from this PR description.

Something like this:

const trackingData = trackingDataUrl ? await fetchTrackingData(trackingDataUrl) : undefined;
if (autoplay) await videoInput.play();
res({...});

Fetch the tracking-data sidecar before videoInput.play() so the looping
video does not advance before Camera Kit receives the source (which would
offset replay tracking from the video timeline), and a failed fetch rejects
before playback starts (no orphaned playing video).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gsasson-sc

Copy link
Copy Markdown
Contributor Author

Good catch — agreed, fixed in 5c1c0cd. The tracking data is now fetched before videoInput.play(), so the looping video can't advance before Camera Kit has the source, and a failed fetch rejects before playback starts (no orphaned playing video). Added a test asserting play() isn't called when the fetch fails. This keeps the tracking data required when trackingDataUrl is set, per the PR intent.

@msilivonik-sc msilivonik-sc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks!

@msilivonik-sc msilivonik-sc merged commit 2990cc8 into Snapchat:main Jun 29, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants